To begin our evaluation phase, we implemented the DECIDE Framework, as outlined below:
First, we determined the goals of our evaluation ...
D= Determine the goals :Which presentation (dynamic or static) do users prefer in a tutorial application and what are the best features of each? We also established a second goal for next usability evaluation round (measuring learning post tutorial) E= Explore the questions: What will we observe? What ask in informal interview? Those in our survey! C= Choose the evaluation approach and methods: Parallel prototype testing, Observation of users, Questionnaire/survey,informal post use interview I= Identify the practical issues: our peer group are the users, two prototypes on computers, interview and observation sheets, online survey prepared, time to complete both tutorials for each user D= Decide how to deal with the ethical issues: Not testing on users in the field so not an issue. Online survey is voluntary and names are not required E= Evaluate, analyze, interpret, and present the data: reliability, validity,collect and collate informal observation and interview data and look for themes, issues etc. Collate results of the Survey--which protoype is generally preferred? Are there any concerns regarding icon use, visual layout, clarity of instructions etc.Any pertinent information offered in the comments sections? Apply results to iterative design
Our research question is:
Which presentation (dynamic or static) do users prefer in a tutorial application and what are the best features of each?
Variables we plan to study:
1. Parallel testing of 2 designs to compare tutorial format designs (in order to combine favored elements or eliminate/change some aspect into an updated design prototype); Animated user- controllable video with voice over vs. static text and graphics
2. An online survey using Survey Monkey (COPY OF QUESTIONS BELOW)
3. Observation of users interacting with both designs, take notes (any tossed out comments, appeals for help, problems, successes etc.);
4. An informal open-ended interview afterward regarding reactions
5. We plan (in the future) to measure the learning gains made from the chosen tutorial design for our Virti-cue product. The outcome of this particular parallel testing will allow us to combine design ideas, in an iterative process, and use this prototype to retest with different users, this time for effectiveness for learning. So, each user will use the newly formed tutorial, followed by interaction with the Virti-Cue product and learning outcomes will be measured/assessed.
Survey Monkey Questionnaire:
Thank you for volunteering to provide feedback on your user satisfaction with the Virti-Cue tutorial. We appreciate the time that you have taken to experience Virti-Cue and to let us know about your user experience. The information you provide is invaluable in helping us to create a user-friendly application.
This survey should take no more than ten minutes to complete and all responses are anonymous, unless you would like us to provide you with further information. You may discontinue taking the survey at any time.
Thank you again!
The Virti-Cue Team
You have used both a static (written) and a dynamic version (animated) of the tutorial. Which version do you prefer and why?
Please rate agreement or disagreement with the following statements regarding the animated tutorial:
The tutorial is a reasonable length.
The instructions are clear and easy to follow.
The tutorial clearly outlines the task.
The pacing of the tutorial was appropriate.
The voice is clear and easy to understand.
Comments
Please rate agreement or disagreement with the following statements regarding the written tutorial:
The tutorial is a reasonable length.
The instructions are clear and easy to follow.
The tutorial clearly outlines the task.
Comments
Please rate the tutorials (both static and dynamic) on the following criteria:
Each icon has a clear meaning.
The visual layout is attractive.
Comments
We appreciate the time you have taken to provide feedback on our tutorial formats. Please see one of our team members to receive your reward!
To begin our evaluation phase, we implemented the DECIDE Framework, as outlined below:
First, we determined the goals of our evaluation ...
D= Determine the goals : Which presentation (dynamic or static) do users prefer in a tutorial application and what are the best
E= Explore the questions: What will we observe? What ask in informal interview? Those in our survey!
C= Choose the evaluation approach and methods: Parallel prototype testing, Observation of users, Questionnaire/survey,informal post use interview
I= Identify the practical issues: our peer group are the users, two prototypes on computers, interview and observation sheets, online survey prepared, time to complete both tutorials for each user
D= Decide how to deal with the ethical issues: Not testing on users in the field so not an issue. Online survey is voluntary and names are not required
E= Evaluate, analyze, interpret, and present the data: reliability, validity, collect and collate informal observation and interview data and look for themes, issues etc. Collate results of the Survey--which protoype is generally preferred? Are there any concerns regarding icon use, visual layout, clarity of instructions etc. Any pertinent information offered in the comments sections? Apply results to iterative design
Our research question is:
Which presentation (dynamic or static) do users prefer in a tutorial application and what are the best features of each?
Variables we plan to study:
1. Parallel testing of 2 designs to compare tutorial format designs (in order to combine favored elements or eliminate/change some aspect into an updated design prototype); Animated user- controllable video with voice over vs. static text and graphics
2. An online survey using Survey Monkey (COPY OF QUESTIONS BELOW)
3. Observation of users interacting with both designs, take notes (any tossed out comments, appeals for help, problems, successes etc.);
4. An informal open-ended interview afterward regarding reactions
5. We plan (in the future) to measure the learning gains made from the chosen tutorial design for our Virti-cue product. The outcome of this particular parallel testing will allow us to combine design ideas, in an iterative process, and use this prototype to retest with different users, this time for effectiveness for learning. So, each user will use the newly formed tutorial, followed by interaction with the Virti-Cue product and learning outcomes will be measured/assessed.
Survey Monkey Questionnaire:
Thank you for volunteering to provide feedback on your user satisfaction with the Virti-Cue tutorial. We appreciate the time that you have taken to experience Virti-Cue and to let us know about your user experience. The information you provide is invaluable in helping us to create a user-friendly application.This survey should take no more than ten minutes to complete and all responses are anonymous, unless you would like us to provide you with further information. You may discontinue taking the survey at any time.
Thank you again!
The Virti-Cue Team
You have used both a static (written) and a dynamic version (animated) of the tutorial. Which version do you prefer and why?
Please rate agreement or disagreement with the following statements regarding the animated tutorial:
Please rate agreement or disagreement with the following statements regarding the written tutorial:
Please rate the tutorials (both static and dynamic) on the following criteria:
We appreciate the time you have taken to provide feedback on our tutorial formats. Please see one of our team members to receive your reward!
http://www.surveymonkey.com/s/K2PD7ZF